Learning Inter-Modal Correspondence and Phenotypes From Multi-Modal Electronic Health Records

Deep Multi-Modal Image Correspondence Learning

Inference of correspondences between images from different modalities is an extremely important perceptual ability that enables humans to understand and recognize crossmodal concepts. In this paper, we consider an instance of this problem that involves matching photographs of building interiors with their corresponding floorplan. This is a particularly challenging problem because a floorplan, a...

Importance of multi-modal approaches to effectively identify cataract cases from electronic health records

OBJECTIVE There is increasing interest in using electronic health records (EHRs) to identify subjects for genomic association studies, due in part to the availability of large amounts of clinical data and the expected cost efficiencies of subject identification. We describe the construction and validation of an EHR-based algorithm to identify subjects with age-related cataracts. MATERIALS AND...

Learning Multi-modal Similarity

In many applications involving multi-media data, the definition of similarity between items is integral to several key tasks, e.g., nearest-neighbor retrieval, classification, and recommendation. Data in such regimes typically exhibits multiple modalities, such as acoustic and visual content of video. Integrating such heterogeneous data to form a holistic similarity space is therefore a key cha...

Inter-modal Learning and Object Concept Acquisition

Learning Concept Taxonomies from Multi-modal Data

We study the problem of automatically building hypernym taxonomies from textual and visual data. Previous works in taxonomy induction generally ignore the increasingly prominent visual data, which encode important perceptual semantics. Instead, we propose a probabilistic model for taxonomy induction by jointly leveraging text and images. To avoid hand-crafted feature engineering, we design end-...